User interface for software applications

ABSTRACT

A user interface for software applications enables the full use of graphical user interface (GUI) software, particularly engineering design software, by visually impaired individuals. The interface combines tactile representations of graphical elements, non-visual cues, and a hardware element to allow the effective placement and interconnection of graphic elements using design software.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. provisional patentapplication Ser. No. 61/002,873, entitled “Multifaceted User Interfacefor Software Applications,” filed on Nov. 13, 2007. The completedisclosure of such provisional patent application is incorporated byreference herein.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with U.S. Government support under the terms ofgrant numbers 0405382 and 0533208 from the National Science Foundation.The U.S. Government has certain rights in the invention.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to computer interfaces, in particular totactile-based user interfaces suited for use by visually impairedcomputer users, and more particularly for use by visually impairedengineers and engineering students in connection with engineering designsoftware.

2. Brief Description of the Related Art

The ability to use the techniques, skills, and modern engineering toolsnecessary for engineering practice is an important requirement forengineering student education and graduation. Germane to this outcome isthe use of simulation software, for such software represents a keyengineering tool that students must master to practice engineering inthe modern world. For example, chemical engineers use process simulationsoftware to solve material and energy balances for proposedmanufacturing schemes, or to analyze current processes for theapplication of possible improvements (i.e., “de-bottlenecking”).Examples of software used by chemical engineers for this purpose includePRO/II by Simulation Sciences, Inc.; ASPEN by ASPEN Technology, Inc.;and CHEMCAD by Chemstations, Inc. Each of these software packagesfunction in a generally similar fashion, and familiarity with oneconfers a transferable knowledge base to operate another, provided thatthe basic input/output (I/O) routines are understood. Choices ofcomponents and flow rates, thermodynamic methods, unit operations, andunit connectivity are provided by the user as input, and the calculationengine then performs the material and energy balance, sizes or ratesequipment, and in some cases provides an estimate of the capitalinvestment associated with a design or improvement.

Recent advances in simulation software design have moved from text-based(i.e., keyword and programming language code) input to Graphical UserInterfaces (GUIs). GUIs are exceptionally attractive to professorsbecause they allow for a significant time savings during studenttraining. Simply put, classroom exercises are freed of debugging thekeyword syntax required to run the calculation engine. Since pointingand clicking generates the code, presumably more time is spentunderstanding the chemical process being studied and how it may becorrectly designed or improved. For example, the simulation of afacility to produce ethylene oxide described in the popular designtextbook Analysis, Synthesis, and Design of Chemical Processes may bebuilt with a GUI by dragging twenty-three appropriate unit operationicons to a design palette, indicating connectivity with lines betweenicons, filling in appropriate drop-down menus, and then pushing the runbutton. This generates approximately 200 lines of keyword code byPRO/II, for example, which is then processed through the calculationengine. Entry of this code by hand would take considerably longer thanimplementing this design using the GUI provided by the software.

One may consider, however, the difficulty associated with using a GUI ifa person has limited or no vision. Classroom activities must obviouslyaccommodate this student in order for the student to succeed. Twoavenues exist for student accommodation: (i) falling back, so to speak,to the exclusive use of keyword files, or (ii) adapting the I/O routineof the software package. The inventors of the present invention believethat completely circumventing the use of a GUI may not be a viablelong-term option for a visually impaired student. The overallrequirement of providing instruction for all design students makes theuse of GUIs invaluable. It is the inventors' understanding, however,that in order for a federally funded university to be in compliance withsection 504 of the Vocational Rehabilitations Act of 1973 (Public Law93-112), accommodations must be made for all qualified students.Although the visually impaired student could work solely with text-basedcode, the quality of modern instruction would be sorely compromised forseveral reasons. First, the time allotted to creative exercises would bediminished to allow for the writing and debugging of keyword files.Second, avoiding GUIs conflicts with the appropriate training outcome.Third, the creation of a separate and distinct educational track for thevisually impaired student only delays the inevitable difficulties thatwill faced by the visually impaired engineering student when thatstudent faces real-world engineering tasks as a practicing engineer.Since the visually impaired student would function in a group settingwith sighted team members, a disparity would exist simply becausesighted engineers and visually impaired engineers would approach processsimulation differently. Those able to employ a GUI would arguably trymore possible solutions, whereas those constrained to keyword code mayexamine fewer scenarios, perhaps resulting in a sub-optimal solution toa particular engineering problem. Such scenarios are clearly notacceptable in modern chemical engineering education and practice.

The art includes a number of attempts to make various types of computersoftware packages usable by those persons with impaired vision. Inparticular, there are several methods used by educators of the visuallyimpaired to convey scientific or mathematical content using software.These techniques typically require some visual acuity to producematerial for the student. The “one-way” techniques include complicatedand costly methods, such as Pictures In A Flash (PIAF), provided byQuantum Technology of Sydney, Australia, and more simple methods, suchas raised-line drawing kits, which are provided through numerous sourcesincluding Maxi-Aids, Inc. of Farmingdale, N.Y. PIAF, or “toasting” as itis routinely called, is a technique that transfers material from amaster to swell paper using heat. First, an image is produced frommaterial, be it handmade or graphic printout. The image is thentransferred using a simple copy machine onto the swell paper, thenraised by exposing the copy to a concentrated light source. PIAF is morecostly because of the paper and light box, but can quickly preparematerial for a student. At the opposite end of the cost spectrum is theraised-line drawing kit. Using this technique, an image is transferredto a thin acetate film by pressure. A stylus similar to those packagedwith personal digital assistants (PDAs) is used to score the film,producing a line. PIAF and similar methods are considered “one way” inas much as a person with visual acuity is required and because they arenot interactive; static representations of material (like graphs ordiagrams) are produced for a student to interpret, but there is no meansby which the student may manipulate those materials directly.

Screen-reading software is a second technique for allowing the visuallyimpaired to interact with computer software. Although digital access hasevolved and developed over the years, the basic idea of usingscreen-reading software dates back to the days of early personalcomputing, and has remained generally the same since that time. Screenreaders have long existed (examples include TextTalker for the Apple II,and VocalEyes for personal computers running the Microsoft DOS operatingsystem) that were able to vocalize the screen content as new informationwas printed to it in a line-by-line fashion. Since an application onlyhad text strings and numerical values as input or output, manipulatinginformation was relatively easy. Though graphics did exist and limitedGUIs were available, the main media by which information was conveyedwas text based, due to the limited computer power that was thenavailable.

With faster processors, more memory, and cheaper storage media, GUIsbecame more profitable to build, maintain, and run on desktop computers.Pointing and clicking is the current standard by which I/O is achieved.Microsoft introduced the Windows operating system, generating selectionpressure towards the creation of “attractive” software relying on GUIs.Since visually impaired users were required to manipulate GUIs, utilityprograms were created that could coherently read graphical informationon the screen. Starting in 1997, Microsoft's Active Accessibility (MSAA)team built and continues to build special tools, APIs (ApplicationProgramming Interfaces), and accessibility standards into the Windowsoperating system that allow leading screen reading packages (such asJaws for Windows from Freedom Scientific, Inc., and WindowEyes fromGwMicro, Inc.) to provide computing accessibility to visually impairedprofessionals. Unfortunately, the architecture of many GUI-basedtechnical programs does not employ MSAA features. This creates a dilemmafor visually impaired users. When one launches an application usingnon-standard controls (i.e., graphics, or unlabelled objects andbuttons) that the screen reader cannot recognize or interpret via MSAAor through the Windows API, the blind person is plunged back intoilliteracy. Such is the case, for example, with standard plantsimulation software used by chemical engineers, such as CHEMCAD, ASPEN,or PRO/II. Although screen readers can interpret some aspects of theinterface, many times there are crucial parts of the GUI that cannot bemanipulated with screen-reading programs. One critical aspect of theseinterfaces includes connecting unit operations via point, click, and/ordrag. These operations are essential for effective use of such software.

Another attempt to make software usable by those persons with visualimpairment is by the use of imbedded audio cues. Non-visual cues,including audible cues may be imbedded in the software to mimic eventsnormally indicated by visual cues, such as textual elements like acursor shape change. Non-visual cues may include audible cues, such asspoken words or sounds, vibrations, kinetic, or any other cue that maybe sensed. This strategy requires a willingness on the part of thesoftware developer to adapt the software package. For users with onlylimited vision, the accurate location of screen items is difficult or insome cases impossible. To overcome these difficulties, distinct soundsmay be introduced. For example, consider when a user is trying toindicate a stream connection from Unit A to Unit B. Conceptually, theuser moves the cursor around the icon of Unit A and encounters anavailable port. When the port is found, a sound can be played toindicate successful capture of the port at either Unit A or Unit B.Sounds may also be used to indicate other functions as well. With thefeedback of sounds, users with limited vision may be promptly informedof events that screen reading programs cannot indicate. The problem withthis approach is that it requires the publisher of the software to imbedthe audio cues in the software itself, making the adaptation of thesoftware for the visually impaired difficult if this effort was not madewhen the software was written.

Finally, tactile representations may be used to locate and place GUIobjects. Tactile representations may be as simple as a small stickerplaced on a transparency or as complex as a stenciled analog of the iconbeing represented. A stencil may be designed to mimic the same size ofthe onscreen icon and provide the user with the needed level of detailto indicate connectivity. Without tactile representations, the inventorsbelieve it is almost impossible to design a facility using the drop andclick features of a GUI. Tactile representations alone, however, arebelieved to be insufficient to provide a complete representation ofsoftware functionality to facilitate effective use by a person who isvisually impaired.

References mentioned in this background section are not admitted to beprior art with respect to the present invention. The limitations of theprior art are overcome by the present invention as described below.

BRIEF SUMMARY OF THE INVENTION

The inventors set forth with the goal of minimizing the differencesbetween visually impaired and sighted persons with respect to their useof software, and in particular with the goal of assisting visuallyimpaired engineering students in the use of modern engineering software.The inventors have achieved this goal through a combination oftechniques that adapt I/O in a creative fashion. The invention resultsfrom a combination of screen-reading programs, non-visual cues, andtactile representations of GUI objects to provide access to modernsimulation software for, in one preferred embodiment, a chemicalengineering student. To the inventors' knowledge, this multifacetedapproach has not been attempted with technical software packages.

It is therefore the goal of the present invention to provide aninterface that allows a person with visual impairment to utilize modernsoftware in a manner most closely analogous to the manner in which thesoftware may be employed by a sighted person using GUIs.

These and other features, objects and advantages of the presentinvention will become better understood from a consideration of thefollowing detailed description of the preferred embodiments and appendedclaim, in conjunction with the drawings as described following:

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a photograph of a tablet personal computer (PC) with a designmedium layer according to a preferred embodiment of the presentinvention.

FIG. 2 is a screen image of a design for hydrocarbon fractionation builtwith a tactile-GUI according to a preferred embodiment of the presentinvention.

FIG. 3 is a tactile representation of a distillation icon according to apreferred embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

In order to examine the effectiveness of combining screen reading,imbedded audio cues, and tactile representations to allow a visuallyimpaired person to operate a GUI, the inventors utilized CHEMCAD version5.5 in a preferred embodiment of the present invention. Although onescript file for JAWS was written to provide for the reading of certainpop-up information boxes, most of the GUI features of CHEMCAD wereaccessed, or initiated, via the adaptations as described herein. Atablet PC, slate model M1200 available from Motion Computing of Austin,Tex., and equipped with a USB-connected keyboard, was employed in apreferred embodiment of the present invention. The visually impairedstudent used this system to design various chemical manufacturingschemes and several separation systems for a hydrocarbon mix.

As stated previously, chemical engineers today use plant simulationsoftware that has a great dependency on GUIs. Although screen readerscan inform the user of certain onscreen changes, automatically move themouse to a predefined point, and duplicate certain mouse actions, it isimpossible to tell the screen reader what to do or where to click ifthere is no a priori spatial knowledge. Such is the case when one istasked with the design of a chemical plant in any GUI environment. Asthe screen reader cannot manipulate graphics alone, a method was neededto solve this problem.

At the center of the solution presented by the preferred embodiment ofthe present invention was the use of tactile representations layeredabove a hardware surface. Two options were considered for this element:an Iveo touchpad from ViewPlus Technologies, Inc. and a tablet PC. Thesetwo options were considered because each represented different means bywhich the pointing device is tracked, and point-and-click operationsoccur. Both options can register interactions at spatially distinctpoints on the surface. The Iveo touchpad's pointing device is the user'sfinger, while the table PC's pointing device is a special pen. With theIveo touchpad, the user interacts with the computer via a USB-connected,letter-sized paper touchpad. When the user touches the touchpad, a mouseclick occurs and is registered spatially. Outside of native Iveoapplications, the inventors hereof found that the touchpad does noteasily allow for drag-and-drop or selection using a mouse click plussweep of an area. These shortcomings made design with the Iveo touchpaddifficult with the native, current drivers provided with the touchpad.

Use of the tablet PC 10, as shown in FIG. 1, was more successful, and istherefore employed in the preferred embodiment of the present invention.The tablet PC registered interactions such as a mouse click not with afinger touch, but with the use of a special pen. This allowed thevisually impaired student to use his or her fingers to “explore” thetactile representations and connection points without spurious clicks ofthe mouse. Moreover, both a right click and left click were possibleusing the pointing device, in this case, a pen. When the pen was withinapproximately 1 cm of the surface of the tablet screen, the pointerhovered over the pallet or finished design, allowing the user to performcertain actions. Specific to CHEMCAD, the user had the ability to hoverin order to read various pop-up windows describing conditions such asflows, temperatures, pressures, and/or sizing of the plant componentsunder the elevated pen point. The visually impaired student using ascreen reader could easily read these pop-up windows much like his orher sighted counterparts, provided that a JAWS script file designed forthis purpose was installed. Once the visually impaired design studenthad a tactile layout of the plant on a removable overlay 11, he or shecould easily find and manipulate tactile representations 12, as shown inFIG. 1. Assuming the tactile representations 12 did not move, thestudent could easily find the tactile markers and perform variousactions related to design and optimization of the process. Tactilerepresentations may be made of various materials, but are preferablyrelatively inflexible and able to support embossing, hole-punching andthe like. In a preferred embodiment, the tactile representations aremade from a heavy acetate polymer with a second layer adhered thereto.

Layout of the plant via the building of the flowsheet involved two majortasks: (i) placing tactile representations of unit operations on thedesign medium, and (ii) indicating and connecting related unitoperations with mouse actions within the tablet PC. This operation isdepicted in FIG. 2, showing layout 14, as displayed on a screen forsighted persons. Audible cues embedded in CHEMCAD address the second ofthe tasks listed above. A comparison between sighted users and thosewith impaired vision can be made to explain the audible enhancements. Asighted user first decides where to make a connection by placing thecursor close to a port on an icon. The user then starts drawing thestream with a left click. Drawing continues until the cursor reaches adifferent type of port on another icon, and finally, the user connectsthe stream with a left click. In contrast, the visually impaired usermust first locate the connection port on the tactile representation. Theuser then listens for the appropriate verbal indicator that signifieseither an inlet or outlet port before performing a mouse click with thetablet PC pen.

Initially, reinforcement stickers such as those typically used tostrengthen paper around a binder punch-hole were used as tactilerepresentations of unit operations or icons. These stickers areadhesive, making design somewhat intuitive to the user. Since thesereinforcement stickers have holes in the center, there is a convenientspot that acts as a tactile irregularity to indicate where the clickcould or should occur, that is, a landing or drop point for the icon.With the auditory feedback mechanisms built into CHEMCAD, thereinforcement sticker method worked well to connect small unit operationicons such as pumps, feeds, and products, since the drop point for aunit, and the feed and product ports are close together. With thestickers, one only had to search for the sound cue for an inlet or anoutlet and drop the pen to click and subsequently place the icon on thescreen.

The reinforcement sticker method was less effective, however, when morecomplicated unit operation icons such as mixers, distillation columns,or heat exchangers were incorporated into a design. With distillationcolumns, for example, there were several points where the cursor may beplaced to make connections. A sticker with one central point did notachieve fine enough tactile control to allow a distillation column to beattached to a feed point and then discharge the bottoms and the tops ofthe column into two different parts of the plant. This shortcomingspurred the development of tactile stencils having multiple tactileirregularities for design of more complex process flow diagrams andcomputer simulations. A variety of tactile irregularities may be used,including holes, embossed patterns or depressions, bumps, and the like.

In the preferred embodiment, complex tactile stencils were designed tomimic the size of the onscreen icons, as shown in FIG. 3. As aconvention, all tactile stencils 16 had a drop point or orientationmarker 18 dedicated to the upper left-hand corner. Stencils 16 wereprinted out by using a 1:1 screen shot of the main pallet taken fromCHEMCAD, sized to the screen of the tablet PC 10, and printed ontoadhesive label paper. A careful mapping was made of the click actionsassociated with the icon drop point 18 and inlet and outlet ports 20.This was accomplished by overlaying a grid using a built-in CHEMCADfunction. Stencils 16 were printed out on adhesive paper and placed on asturdy material. A heavy acetate polymer that is relatively inflexiblehad been used as a backing for the tactile stencils. After the stickerswere affixed to the backing material, holes were punched where the droppoint 18 and ports 20 were indicated. Other tactile irregularitiesbesides holes may be used, including an embossed pattern or depressionor a bump in the tactile representation. Finally, Braille labels 22 wereoptionally embossed on the tactile stencil using a Perkins Brailler. Theupper-right corner of the stencil was then removed to create orientationmarker 24, allowing for proper orientation of the stencil on the screenoverlay 11.

Combining the Braille and alphabetic print on stencil 16 wasadvantageous for several reasons. Notably, it enabled a sighted peer tosee what the visually impaired student had placed on the design medium,thereby allowing for an initial troubleshoot of the design before it wastransferred to the PC. A sighted person was able to fully interact withhis or her visually impaired counterpart because both understood whatthe icon represented, in contrast to exclusively labeling the variouselements with Braille. Alternately, the screen output could be mirroredto an external monitor using a projector, as depicted in FIG. 2.

Other problems that had to be solved included the choice of adhesive anddesign medium or overlay that was layered above the tablet PC screen 10.Since the visually impaired individual must rely on the fact that thedrop points and associated inlets and outlets were in the same place onthe screen every time to allow accurate connection of internal streams,it was vital to secure each stencil 16 onto the design medium 11.Double-sided tape was found to be the best choice for adhesion ofstencils 12 to design medium 11 in the preferred embodiment. Transparentfilm was used for design medium 11 in order to allow a sightedindividual to see through to the screen. Other suitable overlays mayinclude transparencies (commonly made of cellulose acetate) of the sortcommonly used with overhead projectors, polyacetate, or similarlytransparent material. The overlay may fit within the tablet screensurface or may encompass the entirety of the tablet or other screen.

Affixed to all design mediums 11 were marked locations 26, as shown inFIG. 1, for two very important icons not accessible through menucommands: (i) the select unit op submenu icon and (ii) the stream editoricon. A click on the tactile representation of the select unit opsubmenu allowed the user to invoke a submenu that was navigated by thescreen reader. This submenu allowed the visually impaired individual toselect the unit operation icon of choice without having to search thecluttered visual pallet or place too many tactile elements on thetransparency, creating confusion for the student. Another tactile dotmarked the stream editor utility that provides connectivity and stitchesthe process together. These two marker locations 26 were representedwith small, felt-covered stickers since a single, simple click invokedthe respective functions.

The main hurdle presented by GUI-based technical software packages isthe fact that the student or professional must manipulate graphics withthe use of mouse actions to create tangible and working designs. Using acombination of imbedded audio cues and tactile representations allows avisually impaired student to operate CHEMCAD, a technical softwarepackage used in academia and industry, in the preferred embodiment ofthe present invention. The adaptations described herein could, inprincipal, be applied to any GUI-based technical package used in aclassroom or professional setting. The visually impaired person mayeasily generate diagrams with an I/O system such as the tablet PC asoutlined herein and within the scope of the present invention. Using anactive design canvas such as the tablet PC according to a preferredembodiment of the present invention, a visually impaired studentfunctions in a manner essentially equivalent to his or her sightedcounterpart in the classroom.

The present invention has been described with reference to certainpreferred and alternative embodiments that are intended to be exemplaryonly and not limiting to the full scope of the present invention as setforth in the appended claims.

1. A method of controlling an engineering design software applicationwith a user interface comprising a touchscreen providing input to theengineering design software application and an overlay removablyfittable onto the touchscreen, wherein the touchscreen comprises aplurality of spatially distinct points and is operable to display aplurality of engineering design elements, the method comprising thesteps of: (a) identifying among the plurality of engineering designelements a first desired engineering design element to be input to anengineering design and displaying the first desired engineering designelement on the touchscreen; (b) locating a first tactile representationcorresponding to the first desired engineering design element, whereinthe first tactile representation is removable from and repositionableupon the touchscreen; (c) placing the first tactile representation ontothe overlay at a position above a first location where the firstengineering design element is displayed on the touchscreen; (d)identifying among the plurality of engineering design elements a seconddesired engineering design element to be input to the engineering designand displaying the second desired engineering design element on thetouchscreen; (e) locating a second tactile representation correspondingto the second desired engineering design element, wherein the secondtactile representation is removable from and repositionable upon thetouchscreen; (f) placing the second tactile representation onto theoverlay at a position above a second location where the secondengineering design element is displayed on the touchscreen; (g)identifying by touch and responsive non-visual cue an engineering designelement output point on the first tactile representation; (h)identifying by touch and responsive non-visual cue an engineering designelement input point on the second tactile representation; and (i)creating an engineering design by connecting the engineering designelement output point on the first tactile representation and theengineering design element input point on the second tactilerepresentation by means of tactile input to the touchscreen andresponsive non-visual cue wherein a resulting connection is visuallyrepresented on the touchscreen by a displayed line.
 2. The method ofclaim 1, wherein the user interface comprises a pointing device, and thesteps of identifying an output point and identifying an input point andcreating an engineering design are performed by the pointing device. 3.The method of claim 2, wherein said identifying an output point andinput point steps comprise the step of generating audio cues.
 4. Themethod of claim 2, wherein said locating a first and second tactilerepresentation steps comprises the step of reading Braille charactersembossed on a plurality of candidate tactile representations.
 5. Themethod of claim 2, wherein said identifying an output point comprisesthe step of locating a tactile irregularity on the first tactilerepresentation, and said identifying an input point comprises the stepof locating a tactile irregularity on the second tactile representation.6. The method of claim 1, wherein said first and second tactilerepresentations comprise a selectively removable adhesive, wherein saidplacing a first tactile representation step comprises the step ofadhering the first tactile representation to the overlay, and whereinsaid placing a second tactile representation step comprises the step ofadhering the second tactile representation to the overlay.
 7. The methodof claim 2, wherein the first and second tactile representations eachcomprise an orientation marker, and wherein said method furthercomprises the step of orienting the first and second tactilerepresentations on the overlay.
 8. A user interface for an engineeringdesign software application operable to generate a plurality ofgraphical user interface (GUI) elements each corresponding toengineering design elements, the user interface comprising: (a) ahardware surface providing input to the software application, saidhardware surface comprising a touch-sensitive digital image displayscreen comprising a plurality of spatially distinct points; (b) anon-visual cue generated by the software application upon an interactionof a user with one of said spatially distinct points by means of whichthe user is informed whether that one of said spatially distinct pointscorresponds to one of an input and an output; (c) a removable,transparent overlay fittable onto the touch-sensitive digital imagedisplay screen; and (d) a plurality of tactile stencils eachcorresponding to one of the engineering design elements, wherein saidtactile stencils are removable from and repositionable upon theremovable overlay such that the tactile stencils may be positioned abovea corresponding one of the engineering design elements appearing on thetouch-sensitive digital image display screen, wherein each of thetactile stencils further comprise (i) at least one tactile irregularitycorresponding to one of the input and the output; (ii) a touch-readableprint identifying the one of the engineering design elements to whichsuch one of the tactile stencils corresponds, and (iii) avisually-readable print identifying the one of the engineering designelements to which such one of the tactile stencils corresponds.
 9. Theuser interface of claim 8, wherein the touch-sensitive digital imagedisplay screen is operable to display at least one of a plurality ofconnections between at least two of the engineering design elements.